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Long-Term Outcome Study: A Longitudinal Study of 
LA’s BEST Students’ Persistence and Graduation 
Rates 


Deborah La Torre, Seth Leon, Jia Wang, and Li Cai 


CRESST/University of California, Los Angeles 


Executive Summary 


The LA’s BEST Afterschool Enrichment Program is housed at primary centers and 
elementary schools in the Los Angeles Unified School District (LAUSD). The program is free and 
is open to all students in the selected sites on a first-come, first-served basis. At the time of this 
study, LA’s BEST served a student population of approximately 25,000 students at 167 school 
sites. Activities offered at each program site include homework help or tutoring, academic 
enrichment activities in core content areas and technology, and other forms of enrichment such 


as sports or the arts. 


The primary goal of this study was to replicate and improve upon a previous study of 
LA’s BEST participants’ secondary school persistence (or dropout) in light of the program’s 
expansion. Specifically, analyses sought to answer the following research questions: 


e How do the participants and non-participants compare in regards to their 
demographics? Are there differences based on dosage in the LA’s BEST program? 


e How do the participants served in LA’s BEST program sites compare to non- 
participants in their persistence (or dropping out) in school? Are there differences 
based on dosage in the LA’s BEST program? 


e How do participants served in LA’s BEST program sites compare to non-participants 
in regards to their high school graduation? Are there differences based on dosage in 
the LA’s BEST program? 


Study Methodology 


A quasi-experimental design including the use of coarsened exact matching was used to 
establish demographically similar study samples from which valid inferences could be 
generated. The control group was composed of students who attended LA’s BEST schools from 
second to fifth grade, but who had no exposure to the program. Multilevel multiple 
membership classification (MMMC) models were used to examine student outcomes over time. 


This was done to examine long-term impacts on students’ secondary school persistence or 
dropout and high school graduation/completion. Analyses included two cohorts of students 
who were projected to graduate on time from high school in 2016 and 2017. The MMMC 
models used were necessary to account for the nested structure of the data and to take into 


the account student movement between schools. 


Analyses were conducted at two levels. First, we conducted models to examine the 
overall and subgroup results for all students in the study (i.e., the pooled sample) in order to 
ensure sufficient sample sizes to detect intervention effects. We then examined overall and 
subgroup results separately for each of the two cohorts. The dosage-based subgroups used for 
the study were statistically determined in order to account for the overall high level of 
attendance among the treatment students. More specifically, students were in the control 
group if they did not attend LA’s BEST at all during the baseline (second grade) or treatment 
period of the study (third to fifth grade). Students who were included in the treatment group 
were then classified as low (1 to 132 days per year), moderate (133 to 167 days per year), high 
(= 168 days per year), or as treatment interrupted (participated in second grade, but not during 
third, fourth, and/or fifth grade). 


Study Results 


The results of the study generally imply positive long-term impacts on secondary school 
persistence and graduation/completion for students who had higher average attendance in LA’s 
BEST during third through fifth grade. The following presents more details of the findings for 
the study. 


Demographics of the LA’s BEST and Control Students 


e LA’s BEST participants and their statistically matched controls were primarily 
Hispanic and of a low socioeconomic status. More than one-half of the students 
were classified as English language learners. About half of each population and 
sample was female and students generally had lower second grade English language 
arts (ELA) achievement scores than the district mean. 


Secondary School Persistence 


e Multilevel models showed some statistically significant differences between the LA’s 
BEST and control group students concerning secondary school persistence. 


e Significant positive differences were found for the high LA’s BEST attenders when 
compared to the control group. These significant differences were found for the 
analyses of the pooled samples and for the Cohort 2 samples, and approached 
significance for Cohort 1. 


e Significant negative differences were found for the low LA’s BEST attenders when 
compared to the control group. These significant differences were found for the 
analyses of the pooled samples and Cohort 2 samples only. 


e Significant negative differences were found for the LA’s BEST students who had 
interrupted enrollment during third, fourth, and/or fifth grade. These significant 
differences were found for Cohort 1 only, and approached significance for Cohort 2. 


High School Graduation 


e Multilevel models showed some statistically significant differences between the LA’s 
BEST and control group students concerning on time high school graduation or 
completion. 


e Significant positive differences were found for the combined LA’s BEST attenders 
when compared to the control group. These significant differences were found for 
the Cohort 1 samples, and approached significance for the pooled samples. 


e Significant positive differences were found for the high LA’s BEST attenders when 
compared to the control group. These significant differences were found for the 
analyses of the pooled samples and Cohort 1 samples, and approached significance 
for Cohort 2. 


e Significant negative differences were found for the LA’s BEST students who had 
interrupted enrollment during third, fourth, and/or fifth grade. These significant 
differences were found for the analyses of the pooled samples and Cohort 1 
samples, but not for Cohort 2. 


Conclusion 


This study set out to replicate and improve methodologically upon a previous study 
conducted of long-term persistence (or dropping out) outcomes for LA’s BEST participants in 
comparison to a control group of non-participants in the program. The research tracked two 
cohorts of students from second grade to twelfth grade. Despite the differences in 
methodology and in the definition of dosage, the study was able to replicate and extend upon 
the overall finding that students with higher levels of LA’s BEST attendance experienced 
positive benefits regarding secondary school persistence. More specifically, the current study 
found that higher attenders were five percent less likely to dropout and six percent more likely 
to complete high school on time than were the matched control students in the same 
elementary schools who never participated in the program. These results were found within 
the context of more rigorous California State University A-G aligned graduation requirements 
by LAUSD. 


Based on the results of the study, we recommend increased funding for afterschool 


programs to improve attendance among less frequent participants and to increase the number 


vi 


of students served. In large, economically diverse regions such as Los Angeles, even small 
increases in graduation rates may positively impact opportunities for thousands of students and 


their families. 
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Introduction 


Since the publication of A Nation at Risk (Gardner, 1983), which urged adding an hour to 
the school day and 20 to 40 more days to the school year to make the U.S. education system 
more globally competitive, the expansion of student learning time has been an increasingly 
popular policy option for educational scholars and policymakers (Kane, 1994; National Center 
on Education and the Economy, 2008). Recent studies have found that extended learning 
programs, such as afterschool programs, can have positive effects on student outcomes (Bodilly 
& Beckett, 2005; Checkoway et al., 2012; Pedersen, 2012), particularly those that utilize 
frequent assessment (Bodilly & Beckett, 2005), those that focus on high needs students 
(Checkoway et al., 2012), and those in which participants receive adequate dosage (Huang, 
Leon, & La Torre, 2017; Huang, Leon, & La Torre Matrundola, 2014). 


For more than 20 years, researchers and stakeholders have viewed afterschool 
programs as an opportunity to improve academic outcomes (Hollister, 2003), which is an 
important student level factor that can impact persistence and graduation (Goldschmidt & 
Wang, 1999). The providing of afterschool opportunities is considered particularly important for 
students who attend low-income and/or low-performing schools (Afterschool Alliance, 2003; 
Mufioz, 2002). As such, federal and state funding programs, such as the 21* Century 
Community Learning Centers (21°* CCLC) and the After School Education & Safety (ASES) 
Program, have placed greater emphasis on the need for programs to provide homework help, 
supports for core academic subjects, as well as other forms of enrichment such as sports or the 


arts. 


In this study, we provide evidence for two critical issues concerning school level 
outcomes for participants in afterschool programs. Building on previous work conducted by the 
National Center for Research on Evaluation, Standards, and Student Testing (CRESST) at the 
University of California, Los Angeles (UCLA), we measure the long-term impact of the LA’s BEST 


Afterschool Enrichment Program on secondary school persistence as well as on time high 


school graduation or completion. 


Background 


In this section, we explore the literature on afterschool research and evaluation with a 
focus on afterschool attendance. We also present literature on secondary school persistence, 
dropout, and graduation in order to lay the foundation for the student and school level factors 
taken into account during the study. Details of the applicable studies and findings for the LA’s 


BEST program will be provided in the following section about the study context. 
Secondary School Persistence and High School Graduation 


One of the newer requirements of the Elementary and Secondary Education Act of 
1965, as amended by the Every Student Succeeds Act (2015) involves the calculation and 
reporting of high school graduation rates as a measure of school success as well as college and 
career readiness (see U.S. Department of Education, 2017). For the purposes of these 
regulations, students are expected to graduate or complete high school, if they have a 
significant cognitive disability, within four school years of entering ninth grade (U.S. 
Department of Education, 2017). 


Despite the increased emphasis being made concerning this issue, and clear 
improvements overall and across subgroups in recent years, on time graduation or completion 
is still a problem in California. According to the Public Policy Institute of California (2018), 
overall graduation rates increased by 8% and subgroup results increased from 12% to 16% 
between the 2009-2010 and 2015-2016 school years. Despite this, their data also shows that 
the state’s graduation rate stood at only 83% overall and was in the low to mid-seventies for 
major subgroups including Latinos, African Americans, students who are socioeconomically 
disadvantaged, and those classified as English learners (Public Policy Institute of California, 
2018). Results provided by the California Department of Education (2018) for the 2016-2017 
school year shows very similar results with an overall statewide total of 82.7%. Similarly, 
according to the California School Dashboard (California Department of Education, 2017), the 
graduation rates for Los Angeles Unified School District (LAUSD), which hosts LA’s BEST, stood 
at 80.5% during the 2015-2016 school year and at 84.2% for the subsequent year. 


While there is a dearth of research directly assessing the impact of afterschool programs 
on high school graduation rates, research has been conducted on multiple issues that should 
relate, and which were taken into account during the statistical matching process for this study. 


For example, there is some evidence that students who participate in elementary school-based 


programs have better school attendance than those who are in a control group, whether 
statistically matched as in the study conducted of 21° CCLC and ASES programs funded in 
California (Huang & Wang, 2012a) or a recent study conducted of LA’s BEST (La Torre, Leon, 
Wang, & Cai, 2018), or when using unmatched samples (Dynarski et al., 2003; Hartmann, Good, 
& Edmunds, 2011; Jensen et al., 2018). Furthermore, some studies and meta-analyses have 
found positive impacts of afterschool participation in one or both of the key content areas of 
English language arts (ELA) and mathematics (Dynarski et al., 2003; Dynarski et al., 2004; Falls, 
2013; Herrera, Linden, Arbreton, & Grossman, 2011; Lauer et al., 2006; Miller, 2003; O’Donnell 
& Kirkner, 2014; Wilson, 2016). 


While also limited in scope, some research has also been done on the relationship 
between participation in extracurricular activities and secondary school persistence. Fashola 
and Slavin (1998) reviewed the research conducted on six programs that were specifically 
geared towards dropout prevention or encouraging college attendance. While the authors did 
make broad statements about the features of programs that led to success, the methodologies 
used varied greatly. For example, while one study looked at whether students were gone from 
school for 20 or more days while another reported on changes in the percentage of graduates. 
Furthermore, only one of the studies used statistically matched samples—in this case of 
participants and non-participants in the same schools—in order to be able to make causal 
inferences (see Burkheimer, Levinsohn, Koo, & French, 1976; Fashola and Slavin, 1998; 
Burkheimer, Riccobono, & Wisenbaker, 1979). It should be noted, though, that few of the other 
articles reviewed concerning dropout, persistence, or graduation rates for this report noted 


using statistical matching and none noted the use of an experimental design. 


Based on the literature, differences in graduation or persistence rates may be due to a 
variety of reasons. For example, gender, race or ethnicity, and socioeconomic status are all 
common demographic factors that are specifically examined or taken into account. Despite this, 
reports vary in their findings concerning race and gender with some claiming no statistical 
differences (see Marshall, 2017; National Center for Education Statistics, 1998; US Department 
of Education, 2014; Wilson & Tanner-Smith, 2013) and others finding one or more to bea 
significant predictor (Marshall, 2017). In contrast, poverty and socioeconomic status have been 
found to be consistent predictors for dropping out (Cairns, Cairns, & Neckerman, 1989; 
Goldschmidt & Wang, 1999; Rodriguez & Conchas, 2009) and therefore graduation. 
Furthermore, some research has found school level poverty or socioeconomic status to be a 
significant predictor, with lower school level rates relating to lower dropout (Goldschmidt & 
Wang, 1999; Hahn & Danzberger, 1987; Lee, Cornell, Gregory, & Fan, 2011). Based on the 


literature, student characteristics such as school attendance (Allensworth, Gwynne, Moore, & 


de la Torre, 2014; Balfanz, Herzog, & Mac Iver, 2007; Bowers, Sprott, & Taff, 2012; Rumberger, 
2011; Rumberger, 1987), behavior or disciplinary problems (Allensworth et al., 2014; Barnes, 
1992; Bowers et al., 2012; Stearns & Glennie, 2006), and language proficiency (Hahn & 
Danzberger, 1987) can also be predictive of school persistence or dropout. Finally, being 
retained one or more grades has been found to be a consistently strong predictor of secondary 
school dropout (Eide & Showalter, 2001; Goldschmidt & Wang, 1999; Roderick, 1994; Janosz, 
LeBlanc, Boulerice, & Tremblay, 1997; Rumberger, 1995). 


Level of Afterschool Attendance 


The issue of opportunity to learn does not apply only to the issue of school attendance, 
but also to the level of exposure that students receive to homework help, tutoring, and core 
academic content during the afterschool hours. This issue is naturally more confounded during 
the afterschool hours than during the regular school day since the number of days and hours a 
program is offered may vary across or within programs (Vaden-Kiernan et al., 2008) depending 
upon the source of funding and specific circumstances of the school. For example, while 21* 
CCLC and ASES funded programs in California are required to offer services on all school days 
until 69m, participating schools may have different operating hours causing the afterschool 
programs to have different start times. Furthermore, programs receiving funds through these 


two programs may or may not apply for funding to operate during summer intersession. 


Despite the potential importance of accounting for level of attendance as a measure of 
opportunity to learn, few studies examine this issue and those that do vary in how it is 
analyzed. In some studies, such as the randomized control trial of Read 180 Enterprise versus 
the regular afterschool program (Kim, Capotosto, Hartry, & Fitzgerald, 2011), program 
attendance has been used as a continuous predictor variable. In this case, a positive 
relationship was found between greater program attendance and gains in reading vocabulary 


and comprehension when compared to control students. 


Yet other studies have used various ranges of program attendance to create categorical 
grouping variables. These categories seem to be based on theory rather than on empirical 
evidence. For example, a study by Frankel and Daley (2007) of afterschool programs in LAUSD 
created four unequal attendance categories (1-20 days, 21-50 days, 51-100 days, and 101 or 
more days). In this study, the authors found an association between afterschool attendance 
and academic outcomes in ELA and mathematics for elementary students who attended 101 or 
more days per year and for middle school students who participated 51 or more days per year. 


Very similar thresholds were used by Huang and colleagues in their two studies looking at 


attendance dosage and academic outcomes for LA’s BEST (see Huang et al., 2008, 2009, 2014), 


which will be discussed in depth in the following section. 


Huang and Wang (2012a) also took level of attendance into account during their 
longitudinal evaluation of the ASES and 21* CCLC program sites funded in California. While they 
did create attendance categories, the thresholds they set for what constituted regular 
attendance varied by the grade level of the schools being examined. More specifically, they set 
their threshold at 108 days or more per year, equivalent to three or more days per week, for 
elementary school students and at 72 or more days per year, or two or more days per week, for 
middle school students. In this study, while significant positive outcomes on ELA and 
mathematics achievement were not found for regular attenders as a whole for the elementary 
and middle school students, Huang and Wang (2012a) did find positive benefits for regular 
attenders who were African American, classified as special education, or were rated as far 


below basic on the previous year’s standardized achievement test(s). 


The LA’S BEST Afterschool Enrichment Program 


The following section provides information about the LA’s BEST context including its 
history, focus, student demographics, and program structure. This is followed by a discussion of 
the previous evaluation studies of LA’s BEST that included a focus on persistence (or dropping 


out) and students’ motivation to stay in school. 
The LA’s BEST Context 


LA’s BEST seeks to provide a safe haven for at-risk students in neighborhoods where 
gang violence, drugs, and other types of antisocial behaviors are common. The program is 
housed at selected LAUSD elementary schools and is designed for students in kindergarten 
through fifth or sixth grade, depending upon the school. The LA’s BEST sites are chosen based 
on certain criteria, such as low academic performance and their location in low-income, high- 
crime neighborhoods. For optimal program success, and to ensure buy-in from the principals 
and school staff, the school principals have to write an official letter of request for the program 


to be placed in their school site. 


LA’s BEST is a free program open to all students in the selected sites on a first-come, 
first-served basis. Students who sign up for the program are expected to attend five days a 
week in order to reap the full benefits of the program. At the time of this study, LA’s BEST 
served a student population of approximately 25,000 students at 167 school sites. Of this 


population, about 80% were Hispanic and about 12% were African American. Over half of 


students were classified as English learners with the majority of students speaking Spanish as 


their primary language. 


Since its inception in 1988, LA’s BEST has adapted and updated their goals in response 
to educational policies, research, and theory. Over the years, the program has moved past its 
initial emphasis on providing a safe environment and educational enrichment to an emphasis 
on the development of the whole-child. In developmental theory, a whole-child curriculum is 
one that cultivates the development of students’ intellectual, social, and emotional well-being 


so that children can achieve their full potential (Hodgkinson, 2006; Schaps, 2006). 


While individual LA’s BEST sites are given freedom to develop most of their own 
activities, all are required to follow the three and one-half beat structure. The purpose in doing 
so is to meet their grant requirements as well as the program’s mission of providing education, 
enrichment, and recreation. These daily beats, or program segments, include homework help or 
tutoring, academic enrichment in the core content areas and technology, other forms of 
enrichment such as sports or the arts, and a healthy meal. While the individual sites are given 
the freedom to create most of their own lessons and activities, all of the schools also offer the 


KidzLit, KidzMath, and KidzScience curricula, which were designed specifically for use in 


afterschool settings (see http://www.collaborativeclassroom.org). 
Prior Research and Findings on LA’s BEST 


In light of the expansion of the program, LA’s BEST has contracted with CRESST to carry 
out a new study to examine secondary school persistence and high school graduation rates for 
two recent cohorts of LA’s BEST students. To help frame these analyses, we next present details 
about our previous studies conducted in partnership with LA’s BEST about secondary school 
persistence as well as student engagement and academic outcomes, both of which are thought 


to have a relationship with persistence. 


Student engagement. As a follow-up to an earlier qualitative study, Huang and 
colleagues (2007) conducted an exploratory study of the relationship between staff members’ 
and students’ perceptions of social capital. For this study, surveys were administered to 
afterschool site staff and to students in third through fifth grade at 50 LA’s BEST sites. Both staff 
and student surveys included scales to measure staff-student relationships. In addition, the 
staff survey included scales on collective staff efficacy and communication and teamwork, while 
the student instrument also measured student engagement, value of education, and future 
aspirations. When examining student perceptions of staff-student relationships, hierarchical 
linear modeling (HLM) analyses revealed positive relationships with both staff perceptions of 


staff-student relationships and collective staff efficacy. In addition, path analyses revealed a 


positive direct effect of staff-student relationships on student perceptions of the value of 
education as well as indirect positive effects of these relationships on the students’ valuing of 
education, perceptions of school engagement, and future aspirations, as mediated by LA’s BEST 


student engagement. 


Academic outcomes. During the mid-2000s, CRESST conducted a series of quasi- 
experimental design studies that used complementary methodologies to examine longitudinal 
academic outcomes for students as a function of their level of participation in LA’s BEST. The 
first of these studies utilized a residual gain approach to HLM to examine differences in real 
versus predicted growth for students irrespective of their participation in LA’s BEST followed by 
a mixed model approach on the pooled data with a factor to determine if the results varied by 
level of attendance (Huang et al., 2009, 2014). In this study, the issue of self-selection bias was 
addressed through the inclusion and exclusion criteria used for the control and three treatment 
groups as well as through the use of a propensity matching technique. More specifically, 
students were included in the control group if they attended LA’s BEST for 20 or more days 
during the baseline year for the study (second grade) and then did not attend at all during the 
three-year follow-up period (third through fifth grade). In contrast, members of the treatment 
groups attended the program during the follow-up period, but not during the baseline year. 
Students who were included in the treatment were then classified as low (1-20 days per year), 
medium (21-99 days per year), or high/regular attenders (100 or more days per year) using 
afterschool attendance thresholds adapted from those established by Frankel and Daley (2007) 


in their evaluation of afterschool programs in LAUSD. 


The second of these studies on academic outcomes utilized different HLM modeling 
techniques as well as different inclusion and exclusion criteria for the subgroup analyses (Huang 
et al., 2008). In this case, a value-added approach to HLM was used that included growth 
modeling to examine individual student trajectories as well as propensity scores to examine 
differences in growth for the control and three treatment groups based on level of attendance. 
In this case, the low attendance category was treated as a control or reference group (1-20 
days per year) with treatment classified at three levels (21-50 days per year, 51-100 days per 
year, and 101-180 days per year). 


Despite the adaptations made in methodology, these two studies of students who 
participated in LA’s BEST during the mid-2000s showed consistent results (Huang et al., 2008, 
2009, 2014). More specifically, students in the high attending groups for these two studies 
showed significantly better growth trajectories in their mathematics achievement outcomes on 


the California Standards Tests (CSTs) when compared to students who were low attenders or 


who did not attend LA’s BEST at all during the follow-up period. Both studies also revealed that 
neither participation in LA’s BEST nor the level of attendance was significantly related to 
students’ achievement growth in English language arts. Although, it should be noted that ELA 
achievement growth was in the positive direction for the study using the value-added models 
(Huang et al., 2008). 


Secondary school persistence. Huang and colleagues also conducted a quasi- 
experimental design study to examine longitudinal outcomes for students as a function of their 
participation in LA’s BEST (Huang, Kim, Marshall, & Pérez, 2005). This study utilized chi-square 
analyses and Cox regression to examine differences in secondary school persistence between 
four cohorts of LA’s BEST participants (sixth through ninth grade in the 1998-1999 school year) 


and a sample of LAUSD students who did not participate in the program. 


In this study, the issue of self-selection bias was partially addressed through the 
inclusion and exclusion criteria used for the treatment and control groups as well as through 
the use of a statistical matching technique. More specifically, students were eligible for 
inclusion in the control group if they did not participate in LA’s BEST during elementary school. 
In contrast, students were eligible for the treatment group if they participated in LA’s BEST at 
least 20 days in a school year. Students who were included in the treatment were then 
classified as having participated for one year, two years, or three or more years. Despite the 
broad classification of dosage used for the study, statistically lower dropout rates were found 
for the LA’s BEST students who participated in the program for at least three years when 
compared to the matched control sample. In addition, a significant relationship was found 


between days of participation in LA’s BEST and persistence in secondary school. 


Evaluation Questions 


The research questions for the current study are as follows: (1) How do the participants 
and non-participants compare in regards to their demographics? Are there differences based 
on dosage in the LA’s BEST program? (2) How do participants in LA’s BEST program sites 
compare to non-participants in their persistence (or dropping out) in school? Are there 
differences based on dosage in the LA’s BEST program? (3) How do participants served in LA’s 
BEST program sites compare to non-participants in regards to their high school graduation? Are 


there differences based on dosage in the LA’s BEST program? 


Study Methodology 


The study employs a quasi-experimental design consisting of a longitudinal sample of 
academic, behavior, school attendance, afterschool program attendance, and secondary 
persistence and graduation data. The study utilizes two matched samples of LA’s BEST and 
demographically and academically similar students who did not participate in the program who 
were enrolled in second grade during the 2005-2006 or 2006-2007 school years. In addition, 
LA’s BEST students were separated into three groups based on their intensity of attendance in 
the program during the treatment years of third to fifth grade. Once the final samples were 
constructed, multilevel modeling was used to examine outcomes regarding persistence in 


secondary school and high school graduation. 
Sample 


The basis for the sample was composed of existing data gathered by LAUSD and LA’s 
BEST for the 2005-2006 through 2016-2017 school years. The first step in building the sample 
involved generating a sampling frame. This was accomplished by tracking four years of 
elementary school data (i.e., academic, behavior, school attendance, and afterschool program 
attendance) and seven years of secondary school data (i.e., persistence, graduation). In order to 
apply appropriate statistical techniques, all students in the study were required to attend an 
LA’s BEST school during the treatment period. Given that we were examining the relationship 
between program attendance and long-term school persistence and graduation, the study 
periods were defined as baseline (second grade), treatment (third to fifth grade), and follow-up 
(sixth to 12" grade). See Table 1 for further details of the years for each cohort for each study 


period. 


Table 1 


Definition of the Study Periods for the Two Cohorts 


Study period Grades Cohort 1 Years Cohort 2 Years 
BASELINE 
Elementary school Second 2005-2006 2006-2007 
TREATMENT 
Elementary school Third to fifth 2006-2007 to 2008-2009 2007-2008 to 2009- 
2010 
FOLLOW-UP 
Middle school Sixth to eighth 2009-2010 to 2011-2012 2010-2011 to 2012— 
2013 
High school Ninth to 12th 2012-2013 to 2015-2016 2013-2014 to 2016— 
2017 


For any intervention project to demonstrate effects, students must be exposed to 
sufficient treatment. As such, we built on previous studies conducted by CRESST that took into 
account dosage or intensity of participation within LA’s BEST (see Huang et al., 2008, 2009, 
2017; Huang & Wang, 2012a, 2012b). Because of issues with sample size for the lower 
subgroups originally proposed (1-36 and 37-107 days on average per year) as well as the 
varying maximum dosage that students were able to attend the program depending upon 
whether their school offered summer school, it was decided to create a continuous variable to 
represent dosage for each student. Once this was done, three dosage groups of similar size 
were created (see Table 2). In doing so, the research team was able to obtain more fine-grained 
information about the amount of dosage necessary to receive positive growth trajectories in 
comparison to a sample of non-participants. 
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Table 2 


Definition of Study Samples by Dosage 


Sample Definition 


0. Control Attended one or more LA’s BEST schools during the baseline and 
three treatment years, but did not participate in the program 


Treatment 
1. Treatment (low) Participated in LA’s BEST during the baseline and three treatment 
years (1 to 132 days per year) 
2. Treatment (moderate) Participated in LA’s BEST during the baseline and three treatment 
years (133 to 167 days per year) 
3. Treatment (high) Participated in LA’s BEST during the baseline and three treatment 


years (2 168 days per year) 
Treatment interrupted 


4. Treatment (interrupted) Participated in LA’s BEST at baseline (second grade), but did not 
attend the program during one or more treatment years 


As shown in Table 2, the control group for the study was also constructed of students 
who were enrolled in a school that hosted an LA’s BEST site during the baseline (second grade) 
and treatment (third to fifth grade) periods. While this did not account for issues of self- 
selection bias, wherein some families and students may have elected not to participate for a 
variety of reasons, we believe that the use of this control group enabled us to take into account 
within school differences that might account for differences in persistence and 
graduation/completion outcomes during the follow-up period. Since random assignment was 
not possible for this study, coarsened exact matching was employed to control for background 
differences between participating and non-participating students. 


Data 


The study employed a longitudinal sample of existing data sources (see Appendix A for a 
complete list of variables). Elementary school sources included student school attendance, 
behavior data, academic data, and demographics from LAUSD as well as afterschool attendance 
data collected from LA’s BEST. The academic data consisted of the CST scale scores for both 
English language arts and mathematics. Secondary school sources from LAUSD included 


graduation or completion data as well as student withdrawal/dropout data. 
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Analysis Strategy 


The following presents an overview of the descriptive statistics and multilevel models 
used to answer the research questions for the study. Additional details about the outcome 


analysis methodology can be found in Appendix B. 


Descriptive statistics. Descriptive statistics including percentages and means were 
employed to analyze the student background characteristics for the LA’s BEST and non-LA’s 
BEST students. This was done for the populations for each analysis as well as for the overall and 
dosage group samples following statistical matching. Descriptive statistics are also employed to 


examine students’ persistence (dropout) and graduation outcomes. 


Multilevel models for logistic regression. Multilevel models were used to address the 
second and third research questions concerning persistence (dropout) and graduation 
outcomes for LA’s BEST students. Separate models were fit for each comparison (overall, low 
attendance, moderate attendance, and high attendance) as well as for each cohort to 
determine the probability of dropping out or graduating within the typical time-span (2015- 
2016 and 2016-2017, respectively). Additional models were fit to examine outcomes for 
students who attended LA’s BEST at baseline (2005-2006, 2006-2007), but did not attend the 


program at all during one or more of the treatment years (third, fourth, and/or fifth grade). 


To be included in the samples, students were required to attend an LA’s BEST school 
from second through fifth grade, but were allowed to transfer between program schools. In 
addition, our inclusion criteria allowed students to transfer between middle schools and high 
schools during the follow-up periods (sixth to eighth grade, ninth to 12th grade). To allow for 
the inclusion of students who transferred between LA’s BEST schools during the treatment 
period or between secondary schools during the follow-up period we employed multilevel 
regression models with a multiple membership multiple classification (MMMC) scheme 
(Browne, Goldstein, & Rasbash, 2001). MMMC models account for the nonindependence of 
observations within cluster by adjusting the inferences on the parameter estimates for the 
correlations between the responses in a cluster. This modeling approach, however, becomes 
computationally cumbersome using traditional frequentist estimation methods. Because of 
this, as recommended by Browne and colleagues (2001) we employed Bayesian methods using 
Markov Chain Monte Carlo (MCMC) techniques. For our analyses, we also used Benjamini- 
Hochberg corrections to control for false discovery (i.e., false positive or Type | errors) among 
the multiple tests used (see Benjamini & Hochberg, 1995). 


The multilevel models used for these analyses can account for complex classification 


structures, such as a context in which students are nested within schools in each year, but may 
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potentially move between schools during elementary school and/or secondary school. MMMC 
has the flexibility to account for this type of complex nesting structure in which students have a 
one-to-many relationship with schools. In the MMMC modeling approach, each observation at 
the lowest level represents one student. The double arrows linking students to schools signifies 


the possibility of one student being exposed to multiple schools. 


Figure 1 presents the MMMC structure for the regression design that was found to have 
the best fit. In this case, the structure takes into account movement during elementary and/or 
middle school, and not during high school. Appendix B provides more details about the 


selection of this structure for the regression models, including the model fit statistics. 


Elementary and middle schools (3) 


a 


Elementary school (1) Middle school (2) 
Student Student 


Figure 1. Multiple membership multiple classification structure 


1. Each student can map to multiple elementary schools (maximum 3 for the treatment 
period) 

2. Each student can map to multiple middle schools (maximum 3 for the follow-up 
period) 


3. Each student can map to multiple elementary and middle schools (maximum 6 for 
the treatment and follow-up periods) 


Sample Matching Process 


Outcome analyses were conducted for the pooled sample as well as the two separate 
cohorts included in this study to examine secondary school persistence and high school 
graduation/completion. As previously noted, baseline was classified as second grade, treatment 
as third to fifth grade, and follow-up as sixth to 12" grade. In order to be eligible for inclusion in 
the study, students had to attend an LA’s BEST site throughout the baseline and treatment 
periods, have available baseline data, attendance data for the treatment period, and 
dropout/withdrawal and graduation/completion data for the follow-up period. The eligible 


control sample included students who attended one or more schools hosting LA’s BEST during 
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second to fifth grade, but did not participate in the program, and who had available data. 
Achievement and demographic data were used in the matching process. 
Table 3a 


LA’s BEST Study Population: Mobility During the Elementary School Treatment Period 


No mobility Some mobility Total 
LA’s BEST students # % within # % within # % within 
COHORT 1 
LA’s BEST population 1,315 95.0 69 5.0 1,384 100.0 
Low attendance 367 89.5 43 10.5 410 100.0 
Moderate attendance 444 96.1 18 3.9 462 100.0 
High attendance 504 98.4 8 1.6 512 100.0 
Interrupted attendance 1,087 89.2 131 10.8 1,218 100.0 
Total 2,402 92.3 200 Doll 2,602 100.0 
COHORT 2 
LA’s BEST population 1,360 95.4 65 4.6 1,425 100.0 
Low attendance 441 91.9 39 8.1 480 100.0 
Moderate attendance 498 96.0 21 4.0 519 100.0 
High attendance 421 98.8 5 1.2 426 100.0 
Interrupted attendance 1,035 89.2 125 10.8 1,160 100.0 
Total 2,395 92.6 190 7.4 2,585 100.0 


The control samples were selected via a multistage process that eventually created 
matched groups for each of four separate groups of LA’s BEST students. The four groups of LA’s 
BEST students for each cohort included three groups defined by their level of attendance during 
the treatment period (low, moderate, or high), and a group of students who stopped attending 
the program during the treatment period. Table 3a and Table 3b present the eligible samples of 
LA’s BEST and control students for the two cohorts, along with their mobility, during the 


treatment period (third to fifth grade) and middle school (sixth to eighth grade). 
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Table 3b 
LA’s BEST Study Population: Mobility During the Middle School Follow-Up Period 


No mobility Some mobility Total 
LA’s BEST students # % within # % within # % within 
COHORT 1 
LA’s BEST population 1,117 80.7 267 19.3 1,384 100.0 
Low attendance 319 77.8 91 22.2 410 100.0 
Moderate attendance 376 81.4 86 18.6 462 100.0 
High attendance 422 82.4 90 17.6 512 100.0 
Interrupted attendance 971 79.7 247 20.3 1,218 100.0 
Total 2,088 80.2 514 19.8 2,602 100.0 
COHORT 2 
LA’s BEST population 1,024 71.9 401 28.1 1,425 100.0 
Low attendance 331 69.0 149 31.0 480 100.0 
Moderate attendance 388 74.8 131 25.2 519 100.0 
High attendance 305 71.6 121 28.4 426 100.0 
Interrupted attendance 822 70.9 338 29.1 1,160 100.0 
Total 1,846 71.4 739 28.6 2,585 100.0 


The student-level matching technique we employed at baseline (second grade) was 
coarsened exact matching (CEM; lacus, King, & Porro, 2011). CEM is a flexible matching 
approach with many favorable properties, and allows the researcher to specify the precise 
conditions under which students are matched. For categorical variables, such as race/ethnicity 
or poverty, this often entails exact matching, while for continuous measures, such as prior 
individual student achievement and aggregate class level achievement, cut-points for matching 
can be specified. With this approach, we were able to set precise cut-points on the most 
important baseline indicators to ensure that where possible every LA’s BEST student was 
matched with a suitable comparison. Student matching variables we considered in CEM 
included Hispanic, Black, poverty status, female, English language proficiency (English language 
learner, redesignated fluent English proficient), special education status, gifted status, 
achievement in ELA and mathematics, behavior ratings, and prior school attendance. We also 


included an aggregate school level measure of the percentage of students in poverty. 
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In the first stage, we used CEM to align eligible LA’s BEST and control students at 
baseline into equal sized matching strata based on our matching criteria. In this process, we 
were able to match exactly on English language learner, redesignated fluent English proficient, 
special education, and female, and were able to match closely on the continuous variables (e.g., 


achievement in ELA and mathematics, behavior ratings, and school attendance) within strata. 


The next step paired each LA’s BEST student with the closest control student within the 
already created strata. This was done by creating a Mahalanobis distance measure (see 
De Maesschalck, Jouan-Rimbaud, & Massart, 2000) on the non-exactly matched variables, 
sorting LA’s BEST and control students on the distance measure within strata, and then 
matching the LA’s BEST and control students within strata on the sort order of the distance 


measure. 


Descriptive Results 


Table 4a and Table 4b present the student characteristics for the overall LA’s BEST and 
control group samples for two cohorts following statistical matching. Demographics for the low, 
moderate, and high attendance groups for each analytical sample can be found in Appendix C. 
We first discuss the samples prior to matching after which we discuss the background 


demographics following statistical matching. 
Population Demographics 


Before matching, the populations of LA’s BEST and non-LA’s BEST students at baseline 
were similar on many demographic variables. This was true for the populations for both 
Cohort 1 and Cohort 2, which were in second grade during the 2005-2006 and 2006-2007 
school years, respectively. For example, both the LA’s BEST and non-LA’s BEST populations were 
primarily Hispanic (84.0% to 86.6%) and came from families at the poverty level (88.8% to 
94.8%). Approximately half of students were female (48.4% to 54.1%), between one half and 
two thirds of students were classified as English language learners (51.8% to 66.4%), and just 
over five percent were classified as special education (5.5% to 6.0%). Furthermore, students in 
each group attended school close to 160 days per year (155.8 to 162.0), and had mean behavior 


ratings of approximately three. 
Matched Sample Demographics 


For each analytical match, the LA’s BEST and control samples matched very closely. 
Using CEM, we were able to achieve exact matches for female, and classifications as an English 


language learner, redesignated fluent English proficient, or special education. The baseline 
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characteristics presented in Table 4a and Table 4b display the desired baseline equivalence we 
sought to obtain with our matching approach and lend the resulting data samples amenable to 
further analytic methods to examine the LA’s BEST program effects. 

Table 4a 

Demographics of the Cohort 1 LA’s BEST and Control Group Students: Before and After Matching 


Before matching After matching 


LA’s BEST Control LA’s BEST Control 
Student characteristics (N=1,620) (N=4,478) (n=1,384) (n=1,384) 


Race/Ethnicity 


Hispanic (%) 84.0 86.6 84.8 86.7 
Black (%) 8.4 5.0 7.5 5.9 
Asian (%) 3.0 3.2 3.2 2.7 
White (%) 3.0 3.0 2.8 2.9 
Other (%) 1.6 2.2 1.6 1.8 


Special programs status 


Poverty (%) 93.8 94.8 93.8 95.7 
English language learner (%) 60.6 66.4 62.7 62.7 
Redesignated fluent English proficient (%) 0.5 0.6 0.4 0.4 
Special education (%) 6.0 5.9 4.7 4.7 
Gifted (%) 2.0 2.1 2.0 2.0 


Student achievement 
Mean second grade ELA score -0.107 -0.048 -0.073 -0.080 
Mean second grade mathematics score -0.024 0.014 0.002 -0.006 


Other characteristics 


Female (%) 52.2 49.4 51.4 51.4 
Average school attendance 155.8 156.8 157.5 156.8 
Mean behavior rating (1-4) 3.0 3.1 3.1 3.1 
School: Poverty (%) 91.5 91.7 91.7 91.8 
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Table 4b 
Demographics of the Cohort 2 LA’s BEST and Control Group Students: Before and After Matching 


Before matching After matching 


LA’s BEST Control LA’s BEST Control 
Student characteristics (N=1,605) (N=4,110) (n=1,425) (n=1,425) 


Race/Ethnicity 


Hispanic (%) 85.0 86.4 85.7 85.8 
Black (%) 7.8 5.0 7.2 5.9 
Asian (%) 3.2 3.4 3.4 2.7 
White (%) 1.9 3.2 1.8 3.6 
Other (%) 2.2 2.0 1.9 2.0 


Special programs status 


Poverty (%) 88.8 90.7 88.9 91.2 
English language learner (%) 51.8 59.6 54.3 54.3 
Redesignated fluent English proficient (%) 6.2 6.4 5.9 5.9 
Special education (%) 6.0 5.5 3.8 3.8 
Gifted (%) 2.3 2.5 2.5 3.1 


Student achievement 
Mean second grade ELA score -0.066 -0.038 -0.033 -0.023 
Mean second grade mathematics score -0.015 0.062 0.024 0.029 


Other characteristics 


Female (%) 54.1 48.4 53.7 53.7 
Average school attendance 161.6 162.0 163.2 162.5 
Mean behavior rating (1-4) 3.0 3.1 3.1 3.1 
School: Poverty (%) 86.2 87.5 86.2 87.3 


After matching at baseline (2005-2006 and 2006-2007), the two overall LA’s BEST 
student samples were composed largely of students who were Hispanic (84.8%, 85.7%) and of 
low socioeconomic status (93.8%, 88.9%). Regarding special program status, over half of the 
students in each cohort were classified as English language learners (62.7%, 54.3%), with less 
than 10% of students being classified as redesignated fluent English proficient (0.4%, 5.9%), or 
special education (4.7%, 3.8%). In addition, 3.1% or fewer of the LA’s BEST and control students 


were Classified as special education. The CST scale scores were standardized relative to district 
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grade-level performance, based on the district mean and standard deviation. Mean 
performance at baseline for the matched samples was similar to the district-wide performance 
in both ELA and mathematics. When comparing students based on their level of attendance in 
the LA’s BEST program, students with low attendance had lower mean performance on the 
CSTs at baseline, lower school attendance at baseline, and were less likely to be female (see 
Appendix C). Students in the group that had stopped attending LA’s BEST for one or more years 
during the treatment period (third to fifth grade) had baseline characteristics that more closely 
resembled those of students in the low attendance group than students in the higher LA’s BEST 


dosage groups. 


Secondary School Persistence Outcomes 


We examined school persistence (dropout) outcomes for the LA’s BEST participants in 
comparison to the control students who attended schools hosting the program during the 
baseline and treatment periods (second grade, third to fifth grade). First, we present 
descriptive statistics for the overall and dosage group samples. After which we present the 
model results of MMMC analyses for the overall and dosage group samples. Results for both 
sets of analyses are presented for the pooled samples as well as for each cohort (2005-2006 
and 2006-2007). 


Descriptive Statistics for the Matched Samples 


We used the high school withdrawal/dropout records from LAUSD as primary measures 
for the outcome analyses for both cohorts. For this analysis, a student was defined as dropping 
out if they enrolled in LAUSD in ninth grade and subsequently received a leave code that did 


not indicate that they left either the district or the state of California for another school. 


As can be seen in Table 5, there are clear differences in dropout rates across the LA’s 
BEST samples. First, the dropout rates for LA’s BEST students with high attendance are 
consistently lower than the rates for the LA’s BEST students in the low attendance group or the 
interrupted group. This result was found for both cohorts as well as for the pooled sample. 
Second, while the dropout rate for the moderate attendance group was greater than the rate 
for the high attendance group and lower than the rate for the low attendance group, this result 
was only found for Cohort 2 and the pooled sample. With Cohort 1, the moderate attendance 


group had the lowest dropout rate at 12.3%. 


Differences in dropout rates can also be found when comparing the individual LA’s BEST 
samples to the matched comparison samples. For example, students in the high LA’s BEST 


dosage group showed consistently lower dropout rates than did their matched controls. Similar 
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results were found for the matched moderate attendance groups, although in this case the LA’s 
BEST samples only had lower dropout rates for the pooled and Cohort 1 samples, and not for 
the Cohort 2 sample. Finally, as would be expected, the LA’s BEST students in the low 
attendance group and in the interrupted group tended to have higher dropout rates than their 
matched samples. This was true for all analyses except Cohort 2, with the interrupted students 
still having a lower dropout rate than their controls (17.3% to 19.2%). 

Table 5 


Secondary School Dropout Descriptive Statistics 


LA’s BEST Control 
Samples n % n % 
POOLED OUTCOMES 
LA’s BEST vs. combined control (any dosage) 2,809 16.9 2,809 17.4 
Low attendance group 890 21.8 890 17.0 
Moderate attendance group 981 16.7 981 17.5 
High attendance group 938 12.4 938 17.7 
Interruption in attendance (third to fifth grade) 2,378 20.5 2,378 18.9 
COHORT 1 OUTCOMES 
LA’s BEST vs combined control (any dosage) 1,384 15.1 1,384 17.3 
Low attendance group 410 20.5 410 17.3 
Moderate attendance group 462 12.3 462 16.7 
High attendance group 512 13.3 512 18.0 
Interruption in attendance (third to fifth grade) 1,218 23.6 1,218 18.6 
COHORT 2 OUTCOMES 
LA’s BEST vs combined control (any dosage) 1,425 18.6 1,425 17.5 
Low attendance group 480 22.9 480 16.7 
Moderate attendance group 519 20.6 519 18.3 
High attendance group 426 11.3 426 17.4 
Interruption in attendance (third to fifth grade) 1,160 17.3 1,160 19.2 


Note. Lower percentages represent less dropout and greater persistence for the sample. 


Outcome Analysis 


To examine the relationship between afterschool attendance and secondary school 


persistence, we employed an MMMC design. In doing so, we were able to estimate the impact 
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of LA’s BEST attendance on student persistence (or dropout) in high school. This was done for 


two cohorts of students who were projected to complete 12" grade during the 2015-2016 and 


2016-2017 school years. Analyses are also presented for the overall treatment and control 


groups as well as the matched samples by LA’s BEST attendance level during the treatment 


years of third to fifth grade. For these analyses, we followed the recommendations of the What 


Works Clearinghouse. First, we present the Cox index as an effect size for the dichotomous 


outcome (dropped out or persisted in school) and odds-ratios as tests of statistical significance. 


Second, we used Benjamini-Hochberg corrections for multiple comparisons (see Benjamini & 


Hochberg, 1995) for these analyses pooled with the analyses of persistence (or dropout) that 


will be presented in the following section (see Appendix B for more information). 


Table 6 


Secondary School Dropout MMMC Models 


Samples Effect (Cox index) Odds ratio X? (1 DF) 

POOLED OUTCOMES 

LA’s BEST vs combined control (any dosage) -0.019 0.969 0.169 
Low attendance group 0.181 1.349 5.758 
Moderate attendance group -0.034 0.946 0.201 
High attendance group -0.248 0.664 8.868 

Interruption in attendance (third to fifth grade) 0.035 1.059 0.584 

COHORT 1 OUTCOMES 

LA’s BEST vs combined control (any dosage) -0.089 0.863 1.932 
Low attendance group 0.140 1.260 1.543 
Moderate attendance group -0.205 0.713 2.933 
High attendance group -0.210 0.708 3.663 

Interruption in attendance (third to fifth grade) 0.161 1.305 6.690 

COHORT 2 OUTCOMES 

LA’s BEST vs combined control (any dosage) 0.042 1.073 0.483 
Low attendance group 0.246 1.501 5.587 
Moderate attendance group 0.079 1.139 0.622 
High attendance group -0.313 0.596 6.222 

Interruption in attendance (third to fifth grade) -0.127 0.811 3.425 


p value 


0.681 
0.016* 
0.654 
0.003** 
0.445 


0.165 
0.214 
0.087 
0.056 
0.010* 


0.487 
0.018* 
0.430 
0.013* 
0.064 


*o<s.05.** p<.01. 
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Results for the MMMC analyses of high school dropout are presented in Table 6. For 
these analyses, negative effect sizes and odds ratios of less than one indicate lower dropout in 
comparison to the control group, while positive effect sizes and odds ratios exceeding one 
indicate greater dropout. As can be seen, results for the pooled samples indicate a statistically 
significant finding for students in the high attendance group with the LA’s BEST students being 
less likely than their matched controls to drop out of high school (x*(1) = 8.868, p = .003)." In 
addition, a statistically significant effect was found for the low attendance group with the LA’s 
BEST students being more likely to dropout than the students in the control group 
(x? (1) = 5.758, p = .016). 


Results for the cohort level outcomes are also presented in Table 6. In this case, findings 
for Cohort 2 mirrored those found for the overall pooled sample. More specifically, statistically 
significant findings were found with students in the high attendance LA’s BEST sample being 
less likely to dropout (x? (1) = 6.622, p = .013) and the low attendance LA’s BEST students being 
more likely to dropout (x’ (1) = 5.587, p = .018) than were students in their respective control 
samples. While the effect sizes were also in the expected directions for the Cohort 1 high and 
low attendance groups, they did not reach statistical significance. For the Cohort 1 analyses, the 
only statistically significant result was for students in the interrupted group, with LA’s BEST 
students who did not attend the program in third, fourth, and/or fifth grade being more likely 
to dropout than their matched controls (x” (1) = 6.690, p = .010). Model coefficients for the 


MMM C analyses of secondary school persistence/dropout can be found in Appendix D. 


High School Graduation Outcomes 


We examined high school graduation/completion outcomes for the LA’s BEST 
participants in comparison to the control students who attended schools hosting the program 
during the baseline and treatment periods (second grade, third to fifth grade). First, we present 
descriptive statistics for the overall and dosage group samples. After which we present the 
model results of the MMMC analyses for the overall and dosage group samples. Results for 
both sets of analyses are presented for the pooled samples as well as for each cohort (2005- 
2006 and 2006-2007). 


Descriptive Statistics for the Matched Samples 


We used the high school graduation/completion records from LAUSD as primary 


measures for the outcome analyses for both cohorts. As can be seen in Table 7, there are clear 


*The MMMC model results for the pooled high attendance group equate to a predicted probability of dropping out 
of 12.2% for the LA’s BEST group and of 17.1% for the matched control group. 
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trends in the graduation/completion rates amongst the different LA’s BEST samples. In general, 
the graduation/completion rates increased as students were assigned to a higher LA’s BEST 
dosage group. This result was found for the pooled sample as well as the two separate cohorts. 
The only trend that differed among the LA’s BEST samples involved Cohort 1, with the 
graduation rate being equal at 76.6% for both the moderate and high attendance groups. 


Table 7 


High School Graduation/Completion Descriptive Statistics 


LA’s BEST Control 
Samples n % n % 
POOLED OUTCOMES 
LA’s BEST vs combined control (any dosage) 2,809 71.9 2,809 70.0 
Low attendance group 890 64.7 890 67.5 
Moderate attendance group 981 73.3 981 70.6 
High attendance group 938 77.4 938 71.5 
Interruption in attendance (third to fifth grade) 2,378 63.8 2,378 67.4 
COHORT 1 OUTCOMES 
LA’s BEST vs combined control (any dosage) 1,384 73.3 1,384 69.5 
Low attendance group 410 65.4 410 65.4 
Moderate attendance group 462 76.6 462 72.7 
High attendance group 512 76.6 512 69.9 
Interruption in attendance (third to fifth grade) 1,218 60.9 1,218 67.1 
COHORT 2 OUTCOMES 
LA’s BEST vs combined control (any dosage) 1,425 70.7 1,425 70.4 
Low attendance group 480 64.2 480 69.4 
Moderate attendance group 519 70.3 519 68.7 
High attendance group 426 78.4 426 73.5 
Interruption in attendance (third to fifth grade) 1,160 66.7 1,160 67.8 


Differences in high school graduation/completion rates were also found when 
comparing the individual LA’s BEST samples to the matched comparison samples. First, students 
in the overall, moderate attendance, and high attendance groups consistently had higher 


graduation/completion rates than did their matched controls. This was true for the pooled 
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sample as well as for each cohort. Second, LA’s BEST students who had interrupted 
participation in the program during third, fourth, and/or fifth grade had lower 
graduation/completion rates than their matched samples for each analysis. Finally, the LA’s 
BEST students in the low attendance group had the same or lower graduation/completion rates 


than did the matched control students. 
Outcome Analysis 


To examine the relationship between afterschool attendance and secondary school 
graduation/completion, we employed an MMMC design. In doing so, we were able to estimate 
the impact of LA’s BEST attendance on high school graduation/completion. This was done for 
two cohorts of students who were projected to complete 12" grade during the 2015-2016 and 
2016-2017 school years. Analyses are also presented for the overall treatment and control 
groups as well as the matched samples by LA’s BEST attendance level during the treatment 
years of third to fifth grade. For these analyses, we followed the recommendations of the What 
Works Clearinghouse. First, we present the Cox index as an effect size for the dichotomous 
outcome (graduated/completed or failed to graduate/complete on time) and odds-ratios as 
tests of statistical significance. Second, we used Benjamini-Hochberg corrections for multiple 
comparisons (see Benjamini & Hochberg, 1995) for these analyses pooled with the analyses of 
graduation/completion that will be presented in the following section (see Appendix B for more 


information). 


Results for the MMMC analyses of high school graduation/completion are presented in 
Table 8. For these analyses, positive effect sizes and odds ratios of greater than one indicate 
greater likelihood to graduate/complete high school on time in comparison to the control 
group, while negative effect sizes and odds ratios of less than one indicate less likelihood to 
graduate/complete on time. As can be seen, results for the pooled sample indicate a positive 
and statistically significant finding for students in the high attendance group with the LA’s BEST 
students being more likely to graduate/complete high school on time (x7(1) = 8.470, p = .003) 
than their matched controls.” LA’s BEST students in the overall pooled sample were also 
somewhat more likely to graduate/complete on time, although this result did not reach 
statistical significance (p = .071). Finally, LA’s BEST students in the pooled sample who had 
interrupted participation in the program during third, fourth, and/or fifth grade had statistically 


* The MMMC model results for the pooled high attendance group equate to a predicted probability of 
graduating/completing high school on time of 78.4% for the LA’s BEST group and of 72.5% for the matched control 
group. 
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lower graduation/completion rates than did the matched control students (7(1) = 4.799, 
p = .028). 


Table 8 


High School Graduation/Completion MMMC Models 


Samples Effect (Cox index) Odds ratio X?(1DF)  pvalue 

POOLED OUTCOMES 

LA’s BEST vs combined control (any dosage) 0.070 1.123 3.268 0.071 
Low attendance group -0.070 0.891 1.114 0.291 
Moderate attendance group 0.088 1.156 1.799 0.180 
High attendance group 0.203 1.398 8.470 0.003** 

Interruption in attendance (third to fifth grade) -0.088 0.865 4.799 0.028* 

COHORT 1 OUTCOMES 

LA’s BEST vs combined control (any dosage) 0.120 1.219 5.014 0.025* 
Low attendance group -0.000 1.000 0.000 0.992 
Moderate attendance group 0.121 1.221 1.515 0.218 
High attendance group 0.230 1.462 6.048 0.014* 

Interruption in attendance (third to fifth grade) -0.151 0.780 7.640 0.006** 

COHORT 2 OUTCOMES 

LA’s BEST vs combined control (any dosage) 0.024 1.040 0.186 0.666 
Low attendance group -0.155 0.775 2.856 0.091 
Moderate attendance group 0.079 1.140 0.766 0.381 
High attendance group 0.194 1.377 3.248 0.072 

Interruption in attendance (third to fifth grade) 0.007 1.012 0.015 0.903 


*o<s.05.** p<.01. 


Cohort level results are also presented in Table 8. In this case, findings for Cohort 1 
mirrored those found for the overall pooled sample. More specifically, statistically significant 
positive results were found for the LA’s BEST students in the overall sample (x7(1) = 5.014, 

p = .025) and in the high attendance group (x7(1) = 6.048, p = .014). In each case, the LA’s BEST 
students were more likely to graduate/complete on time than were the matched control 
students. In addition, as with the pooled sample, the LA’s BEST students with interrupted 
attendance in the program during third, fourth, and/or fifth grade were statistically less likely to 


graduate on time than were the matched control students (x7(1) = 7.640, p = .006). Finally, 
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while no statistically significant results were found for Cohort 2, the results for the low 
attendance and high attendance groups were in the expected directions. Model coefficients for 


the MMMC analyses of graduation/completion can be found in Appendix D. 


Discussion and Conclusion 


The retrospective data did not allow us to assign students randomly to treatment and 
control conditions. However, a quasi-experimental design including the use of coarsened exact 
matching was used to establish demographically similar study samples from which valid 
inferences could be generated. The control group of students is composed of students in the 
program schools who had no exposure to LA’s BEST during second through fifth grade. 
However, it is possible that control students did attend other afterschool activities in the 
community or at the program schools such as tutoring or the Youth Services After School 
Program, a drop-in program that provides homework help and playground activities at over 600 
elementary and secondary schools in LAUSD (see LAUSD Beyond the Bell Branch, 2018). As 


such, the treatment effects represent lower bound estimates of program effects for LA’s BEST. 


Multiple membership multiple classification models were used to examine the 
likelihood of persisting (or dropping out) as well as graduating/completing high school on time 
(i.e., within four years of entering ninth grade). This was done to examine both cohorts of 
students (2005-2006 and 2006-2007) followed for the study. The models used were necessary 
to account for the nested structure of the data, and to take into account the movement of 
students between schools during elementary and secondary school. It should be noted, though, 
that students were allowed to move between LA’s BEST schools during the baseline (second 
grade) and treatment periods (third to fifth grade), but were not allowed to attend other 
elementary schools that did not host the program. In addition, since the methodology required 
larger sample sizes in order to find significant findings, we were not able to disentangle 


potential differences in effects of the individual LA’s BEST program sites. 


The results of the study generally imply positive long-term effects on secondary school 
persistence and graduation/completion for students who participated in LA’s BEST from second 
through fifth grade. Despite this, statistically significant results were more likely to be found for 
the pooled samples of students who were in second grade during the 2005-2006 and 2006-— 
2007, than for the separate cohorts. The following discusses the findings for each of the 


research questions in greater depth. 
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Demographics of the LA’s BEST and Control Students 


Prior to matching, students in the LA’s BEST and control populations showed some 
demographic differences. More specifically, control students were somewhat more likely to be 
Hispanic and somewhat less likely to be Black than were the students who attended LA’s BEST 
during the treatment period. Students in the control population were also more likely to be 
classified as English language learners during second grade than were the LA’s BEST population. 
In addition, the LA’s BEST population was somewhat more likely to be female than were the 
control population. 


Following coarsened exact matching, we were able to create samples that were 
demographically very similar. More specifically, we were able to match our samples exactly for 
female, English language learner, redesignated fluent English proficient, and special education. 
When examining the two overall cohort samples, the control students were slightly less likely to 
be Black, had slightly lower average school attendance, and were slightly more likely to be in 
poverty than were the matched LA’s BEST students. It should be noted, though, that some 


differences in the trends for these three variables were found for the dosage level samples. 
Long-Term Impacts on Secondary School 


Long-term impacts on secondary school persistence (dropout) and high school 
graduation were examined separately. To isolate treatment effects more readily, the multilevel 
analyses focused first on the pooled outcomes and then on the individual outcomes for two 
cohorts of students who were projected to finish high school in LAUSD during the 2015-2016 
and 2016-2017 school years. It should be noted that the cohorts were pooled for the primary 
analyses in order to improve the overall sample size and to ensure adequate samples for the 


additional models conducted to examine the impact of attendance on the outcomes. 


The results of the pooled samples for both sets of analyses underline the importance of 
greater exposure to the LA’s BEST program on long-term secondary school outcomes. The 
multilevel models we utilized found significant positive effects for the high attendance groups 
in regards to both secondary school persistence and on time graduation or completion of high 
school. More specifically, students who attended the program an average of 168 or more days 
per year during third through fifth grade were 4.9% less likely to drop out and were 5.9% more 
likely to complete high school on time than were the matched control students, who also 
attended schools hosting LA’s BEST during elementary school. In addition, the models revealed 
that the positive impacts disappeared with lower attendance in the program. More specifically, 
LA’s BEST students in the pooled sample who had lower program attendance of one to 132 days 


per year were statistically more likely to drop out of secondary school than were the non-LA’s 
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BEST students. Likewise, students in the pooled sample who had an interruption in enrollment 
in LA’s BEST during third, fourth, and/or fifth grade were significantly less likely to graduate on 


time than were their matched controls. 


While the results were less stable, most likely due to the much smaller sample sizes, the 
cohort level models did provide some additional evidence concerning the importance of 
afterschool attendance dosage. That is, the Cohort 2 outcomes for the models measuring 
secondary school persistence were consistent with the pooled results for the low and high 
attendance groups. Likewise, the Cohort 1 results for on time graduation were consistent with 
the pooled results for the high and interrupted LA’s BEST attenders. It is also interesting to note 
that the model that examined the overall sample of Cohort 1 students found that LA’s BEST 
students were significantly more likely to graduate on time when compared to their overall 


Cohort 1 control group. 
Conclusion 


This study set out to replicate and improve methodologically upon a previous study 
conducted of long-term secondary school persistence for LA’s BEST participants in comparison 
to a control group of non-participants in the program. The research tracked two cohorts of 
students from second grade (2005-2006 and 2006-2007) to 12” grade (2015-2016 and 2016-— 
2017). While all students were required to attend a school hosting the LA’s BEST program 
during second through fifth grade, the models used were able to account for movement 
between LA’s BEST school during this time as well as any movement between middle schools 
that occurred. Unfortunately, due to issues of sample size and model fit, as well as the large 
number of untraditional high schools in the district, including alternative schools that are 
specifically focused on helping students who are highly at-risk for dropping out, we were not 
able to account for movement between schools following eighth grade. Furthermore, due to 
the scope of the study and issues of data access we were not able to account for any additional 
afterschool exposure that the students in either the treatment or control groups might have 
received during their attendance in LAUSD. Nor were we able to conduct analyses at the school 


level in order to determine whether the program had differential effects. 


Despite the stated limitations, the study did find evidence concerning the importance of 
maintaining higher attendance in the LA’s BEST program. More specifically, students who 
attended an average of 168 or more days per year during upper elementary (third to fifth 
grade) were 4.9% less likely to drop out of high school and were 5.9% more likely to graduate in 
four years than were non-participants. Likewise, students with lower attendance seemed to 


lose the benefits of the program with null results for the moderate attendance groups, and null 
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or significantly negative results for LA’s BEST students who had attendance of 132 days or less 
over three years or who had an interruption in enrollment for one to three years. As such, it 
seems important for the program to find ways to motivate lower or moderate attending 


students to stay in the program or increase their level of participation. 


Recommendations for Follow-Up 


While rigorous methods were used for the current study, we believe that further research 
of longitudinal outcomes for LA’s BEST participants would benefit both the program and the 
afterschool field at large, in addition to some more specific studies of LA’s BEST in response to 
the program interest. The following presents our suggestions for studies of contextual factors 
that might play a role in high school persistence and graduation as well as studies of past 
participants college and career trajectories. These suggestions include immediate to long-term 
options ranging from the analysis of existing data to multi-year research and/or development 
projects. 


Chronic absenteeism and school attendance. According to the most recent WWC practice 
guide for dropout prevention, school attendance and behavior variables such as suspensions 
are reliable predictors of students’ secondary school persistence (Allensworth et al., 2014; 
Bowers et al., 2012; Rumberger et al., 2017). For example, Allensworth and colleagues (2012) 
report that middle school attendance is a strong predictor of high school graduation. While 
previous research conducted by CRESST of LA’s BEST (La Torre, Leon, Wang, & Cai, 2018) has 
found a relationship between program attendance and day school attendance, no causal work 
has been completed on the issue. Thus, we propose to use hierarchical models to examine 
longitudinal impacts of LA’s BEST participation on secondary school attendance and behavior. 
Data used for the study would include LA’s BEST and school attendance data, behavior data, 


and demographic information. 


Subgroup analyses regarding persistence and graduation. As previously mentioned, 
graduation rates tend to vary by major subgroups. For example, the California School 
Dashboard (California Department of Education, 2017), shows that while the overall graduation 
rate for LAUSD stood at 84.2% in 2017, only 58.3% of students with disabilities graduated or 
completed high school within four years. As such, we believe that it would be valuable to 
examine the descriptive data for the subgroups to see how they compare with the two sample 
cohorts included in the current study of secondary school persistence and graduation. The 
results could help the program to target their intervention and improve the overall program 


SUCCESS. 
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Social and emotional learning outcomes. One of the student supports recommended by 
the WWC to help prevent secondary school dropout involves the providing of social and 
emotional learning (SEL) and skills training (What Works Clearinghouse, 2017). As such, we 
recommend an examination of climate survey results collected by LAUSD since the 2010-2011 
school year. One potential extension to this work involves an examination of the literature on 
affordances and barriers for the development of these skills and mindsets (e.g., self- 
management, self-efficacy, and growth) with an emphasis on afterschool settings followed by a 
panel meeting of afterschool stakeholders to contextualize the findings. In addition, the work 
could be extended through the conducting of case studies at LA’s BEST sites that report high 
levels of social emotional skills and mindsets. We also advocate the conducting of longer-term 
research relating SEL development within the afterschool setting to student achievement 


outcomes. 


School and community based social capital. From 2004 to 2006, CRESST conducted a 
series of multi method studies looking at the social capital within LA’s BEST. This included 
qualitative case studies at six program sites as well as survey research with students and site 
staff at an additional 50 sites. We propose building on this work through the conducting of one 
or more case studies at LA’s BEST sites to examine the influences of the program on 
participants, the school at large, and/or the local community at two or three of the most 
successful program sites. Some of the potential questions that might be explored include the 
promoting of key outcomes of interest by the program (e.g., social emotional outcomes, school 
attendance, and academics), participants experiences with the program, and influential 
experiences and practices developed by stakeholders that could affect participant outcomes. 
Results of this work could help LA’s BEST to identify and develop their own logic model, share 
their successes with the afterschool field, and contribute to the best practices for afterschool 


programs. 


College and career trajectories of past participants. As noted in the 2008 WWC practice 
guide for dropout prevention, it is important to provide students with the learning and “skills 
needed to graduate and to serve them after they leave school” (Dynarski et al., 2008, pg. 11). In 
this frame, we suggest further longitudinal research to examine college attendance and the 
types of careers pursued by past participants in LA’s BEST. 


During 2017, scholars at the Luskin School of Public Affairs at UCLA completed a study 
looking at college attendance and persistence for graduates of LAUSD (see Foulsham, 2017). In 
light of this, we suggest a study to examine college attendance, the types of schools attended 
(e.g., two-year, four-year institutions), persistence, and/or completions for prior attendees of 
LA’s BEST. As with the current study, we propose to take dosage of participation with LA’s BEST 
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into account to lend statistical weight to our analyses. Existing sources that might be used for 
this study include data from the National Student Clearinghouse; LAUSD data on students’ high 
school performance, students’ climate survey responses regarding college readiness and future 
orientation, and students’ demographic and background data; and LA’s BEST attendance data. 


In light of the afterschool field’s current focus on providing both academic and other 
forms of enrichment, we believe it would also be valuable to examine the career choices of past 
participants. Options for carrying out this work involve the systematic collection of data about 
college majors and careers from public web sites such as LinkedIn and/or the online surveying 
of past participants. Once gathered, a coding scheme would be developed using a subset of the 
data, and then content analyses would be conducted to provide both qualitative information 


and frequencies. 


College and career trajectories of site staff. Based on a recent annual report, LA’s BEST 
(2017) currently employs over 2000 program staff to work with students. Based on the earlier 
work of Huang and colleagues (2007), we know that LA’s BEST staff members often develop 
strong mentoring relationships with students. This is reiterated in one of the quotes highlighted 
in the current annual report, “I thought | was just going to work with kids, but | realize | entered 
a community and found a career” (LA’s BEST, 2017). As with the student study, we suggest 
gathering data using the systematic collection of data from public web sites such as LinkedIn 
and/or the online surveying of current and former staff members. For this work, we also 
suggest the disaggregation of results by job type (e.g., program workers versus leadership 


positions) as well as previous participation in the program during elementary school. 


Staff training and program development. In recent years, funding agencies, such as the 
Institute of Education Sciences (IES) at the U.S. Department of Education, have recognized the 
potential role of social and behavioral contexts for academic learning. As such, we recommend 
the seeking of grant funds for research and development that can build on the already existing 
systems within LA’s BEST. For example, grant funds could be used to train site staff, conduct a 
small-scale intervention study, and then develop a logic model for how to integrate the 


intervention across the program. 
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Appendix A: Data Sources 


Table Ala 


Data Sources for the Elementary School Baseline and Treatment Periods 


Elementary school variables 


Cohort 1 


Cohort 2 


Academic 
CST ELA: Scale scores 
CST mathematics: Scale scores 
Attendance 
LA’s BEST daily attendance 
LAUSD daily attendance 
Behavior 
LAUSD behavior ratings 
Demographics 
Race/Ethnicity 
Demographics 
English proficiency status 
Female 
Gifted 
Grade level 
Parent some college education 
Poverty 
Special education 
Linking variables 
School IDs 


Student researcher IDs (de- 
identified) 


2005-2006 to 2008-2009 
2005-2006 to 2008-2009 


2005-2006 to 2008-2009 
2005-2006 to 2008-2009 


2005-2006 to 2008-2009 
2005-2006 to 2008-2009 
2005-2006 to 2008-2009 


2005-2006 to 2008-2009 
2005-2006 to 2008-2009 
2005-2006 to 2008-2009 
2005-2006 to 2008-2009 
2005-2006 to 2008-2009 
2005-2006 to 2008-2009 
2005-2006 to 2008-2009 


2005-2006 to 2008-2009 


2005-2006 to 2008-2009 


2006-2007 to 2009-2010 
2006-2007 to 2009-2010 


2006-2007 to 2009-2010 
2006-2007 to 2009-2010 


2006-2007 to 2009-2010 
2006-2007 to 2009-2010 
2006-2007 to 2009-2010 


2006-2007 to 2009-2010 
2006-2007 to 2009-2010 
2006-2007 to 2009-2010 
2006-2007 to 2009-2010 
2006-2007 to 2009-2010 
2006-2007 to 2009-2010 
2006-2007 to 2009-2010 


2006-2007 to 2009-2010 


2006-2007 to 2009-2010 


39 


Table A1b 


Data Sources for the Secondary Schoo! Follow-Up Period 


Secondary school variables 


Cohort 1 


Cohort 2 


Graduation/Completer 
Completer codes 
Completer reasons 

Withdrawal/Dropout 
Withdrawal codes 
Withdrawal reasons 
Dropout codes 
Dropout reasons 

Linking variables 
School IDs 


Student researcher IDs (de- 
identified) 


2009-2010 to 2015-2016 
2009-2010 to 2015-2016 


2009-2010 to 2015-2016 
2009-2010 to 2015-2016 
2009-2010 to 2015-2016 
2009-2010 to 2015-2016 


2009-2010 to 2015-2016 


2009-2010 to 2015-2016 


2010-2011 to 2016-2017 
2010-2011 to 2016-2017 


2010-2011 to 2016-2017 
2010-2011 to 2016-2017 
2010-2011 to 2016-2017 
2010-2011 to 2016-2017 


2010-2011 to 2016-2017 


2010-2011 to 2016-2017 
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Appendix B: Outcome Analysis Methodology 


Regression Model Specification 


To be included in the samples, students were required to attend an LA’s BEST school at 
baseline (second grade) and for all three treatment years (third to fifth grade), but were 
allowed to transfer between program schools. In addition, our inclusion criteria allowed 
students to transfer between middle schools and high schools within LAUSD during the follow- 
up periods (sixth to eight grade, ninth to twelfth grade). Because of this complex clustering of 
students within schools, we tested the goodness of fit of different models. It should be noted 
that we chose not to include high schools in any of the models because of the large number of 
untraditional “option” schools in the district including 41 continuation high schools, which are 
specifically designed to serve students who are at risk of not graduating/completing (see Los 
Angeles Unified School District, n.d.). 


We assessed the fit of six different models to account for the clustering of students 
within schools across the elementary and middle school settings. The first set of models 
address mobility at the elementary school level (see Equation 1), the second set at the middle 
school level (see Equation 2), and the third set at both the elementary and middle school levels 
(see Equation 3). A basic two-level multiple membership multiple classification (MMMC) model 
was used to estimate the impacts of the LA’s BEST intervention on students’ secondary school 
persistence and then on their high school graduation. Each of the equations represents an 
extension for dichotomous outcomes of the general specification for the MMMC model used 
for continuous outcomes proposed by Browne and colleagues (2001, equation 6) and applied in 


Tranmer and colleagues (2014, equation 3). 


. I 2 
m= logit) = B+ wu)? +e 
jEElemSchool(i) 
School(i) ¢ (1,...,J) 
: oe ~ N(0, Ouc))» ei ~ N(O, oé) (1) 
tyne, = x i ee 
Ni ogit (1;) xj P a Wit + e; 


j€MiddSchool(i) 


School(i) € (1,...,J) 
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: un ~ N(0, Ouc))» ei ~ N(O, oé) (2) 


. ' 2 2 3 3 
n; = logit(m;) = x;PB + os wou? + SS wus; + e; 


jJ€ElemSchool(i) j€MiddSchool(i) 


ElemSchool(i) ¢ (1,...,J), “a ~ N(O, Oua))» e, ~ N(0,02) 


MiddSchool(i) ¢ (1,....J), 4 ~ N(O,oz@)),  e ~ N(0,03) (3) 


In each of the models (aligned with one of the three equations), 7; is the log-odds for 
the outcome of interest (student persistence or graduation/completion), Xi is a vector of the 
fixed covariates, and £ is the vector of the corresponding fixed effects. 


2) , (2) 


Within the term ) jcztemschnool(i) Wi,jU; > u; “is the set of j random effects for the 


J 
elementary schools included in the selected data set, and w, ; is the weight that sums to 1 for 


each student applied in proportion to the time assigned with each school. 


Within the term ) jemiadschool(i) wi guy ' y 


middle schools included in the selected data set, and w; ; is the weight that sums to 1 for each 


is the set of j random effects for the 


student applied in proportion to the time assigned with each school. 


The two models that address only elementary school clustering (Model 1, Model 2) are 
shown in Equation 1. Similarly, the two models that address only middle school clustering 
(Model 3, Model 4) are shown in Equation 2, and the models that address both elementary and 
middle school clustering (Model 5, Model 6) are shown in Equation 3. We simplify Model 1, 
Model 3, and Model 5 by only selecting the first elementary and or middle school rather than 
accounting for student mobility. As a result, the w; ; term becomes a constant of 1 in these 
three models. Model 1 and Model 3 each become basic two-level models, while Model 5 


becomes a basic cross-classified model. 


In addition to the LA’s BEST treatment indicators, the fixed effects included student 
characteristics to identify the matched control sample of students, such as baseline 
achievement, socioeconomic status, demographics, language proficiency, and participation in 
special education. For each of our outcome analyses, we used a threshold of p < .05 to 
determine whether there was a statistically significant impact of LA’s BEST. We included 
dummy variables to identify each comparison that we tested (e.g., the low LA’s BEST group and 


their matched controls). 
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The following presents an example of the full model for the graduation/completion 
specification from our best fitting model. This is an MMMC model, which accounts for student 


mobility in both the elementary and middle school settings (Model 6). 


log — odds completion; 
= Bo +SchPoverty; * 6, + priorELA; * B, + priorMath; * B3 
+ Behavior; * 6, + DayAttend; +* 6B; + Female; * Bg + LEP, * B7 
+ RFEP; * Bg + Black; * By + Hispanic; * Big + SPED; * By, 
+ Gifted; * By2 + Poverty; * Bj3 + LOowLA'sBEST * By4 
+ LowComp * B,; + ModLA'sBEST * Bi, + ModComp * B17 
+ HiLA'sBEST * B13, + HiComp * Big + InterruptedLA'sBEST 


i B20 a 3 wy + W..U 


ij “Oj ij “Oj 
jJ€ElemSchool(i) jJ€MiddSchool(i) 


ElemSchool(i) ¢ (1,...,J), a ~ N(O, Ou))» e, ~ N(0,02) 
MiddSchool(i) ¢(1,...,J), uy? ~ N(O,2@)),  & ~ N(0,02) 


In this model, log — odds completion, is the log-odds of graduation/completion for 


student /; 


The dummy coded effect indicators—LowLA'sBEST,, ModLA'sBEST; , and 
HiLA'sBEST; —are coded as 1 for students receiving the indicated level of LA’s BEST attendance 
dosage and 0 otherwise. The dummy coded control group indicators—LowComp;, ModComp, , 
and HiComp;—are coded as 1 for control students paired with LA’s BEST students receiving the 
indicated level of LA’s BEST attendance dosage and 0 otherwise. The InterruptedLA'sBEST; 
indicator represents students who attended LA’s BEST at baseline (second grade), but who 
stopped attending during one or more of the treatment years (third, fourth, and/or fifth grade). 
The paired control students for InteruptedLA'sBEST; comprise the reference group in the 


regression model. 


Female;, LEP;, RFEP;, Black;, Hispanic; SPED,, Gifted;,and Poverty; are student 


demographic indicators coded 1 if the status is present and 0 if absent; 


SchPoverty; is the aggregate school level percentage of students who received 


free/reduced school lunch for student i; 


priorELA; , and priorMath, are standardized student CST scores from the baseline 


year (second grade); 
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Behavior; is the aggregated composite mean of five behavior ratings from the baseline 


year (second grade) for student i; 


DayAttend; is the number of days of regular school attendance in the baseline (second 


grade) year for student /; 


B, is the effect of school level poverty; 

Bz & B3 are the effects of the prior score covariates; 
Bh, is the effect of the behavior covariate; 

B; is the effect of the school attendance covariate; 


Be --- B13 are the effects of the demographic covariates; 


We construct our comparison tests from 1,4 ... B29 as follows: 

The effect of Low dosage LA’s BEST attendance is a test of whether 6,4, = [45 

The effect of Moderate dosage LA’s BEST attendance is a test of whether Bi, = (47 
The effect of High dosage LA’s BEST attendance is a test of whether 61, = [49 


B20 is the effect for students who attended LA’s BEST; 


It should be noted that in order to test the effect of the combined LA’s BEST dosage 
group we ran a separate model in which LowLA'sBEST;, ModLA'sBEST,, and HiLA'sBEST; 
were dropped and combined into a single LA’s BEST treatment variable (LA’sBESTd; ), and 
similarly LowComp;, ModComp,, and HiComp; were dropped and combined into a single 
variable of paired control students (Compd; ). In that model we again tested whether the 


groups were equal (6,4, = fis). 


7 ae e; are the error components at the elementary school, middle school, and 


student-levels, respectively, with all assumed to have a mean of 0 and a variance, 
2 2 2 ‘ 
Ou(2) »Fy(3), Nd og respectively. 


Implementing MCMC Estimation 


Within MCMC estimation, there are various options available to produce the most 
appropriate process, to address the quality of estimates, and to select the best fitting model. 


One option is a reparameterization method known as using orthogonal fixed effect vectors. This 
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method is considered most useful for non-normal response models, as is the case in the models 
used for this study (Browne et al., 2001). 


When we evaluated the quality of the mixing in MCMC using the Effective Sample Size 
(ESS) of the fixed and random effects it was clear that estimation was improved when using 
orthogonal fixed effect vectors for the outcomes. In addition, we also evaluated the accuracy 
using diagnostics (see Brooks & Draper, 2007; Raftery & Lewis, 1992), and determined it was 
necessary to set the monitoring chain length to 15,000 iterations. We tested the fit for each of 
the six models and compared the results using the Bayesian Deviance Information Criterion 
(DIC). In each case, the goal was to determine if the MMMC approach provided a better fit than 
a less complex model that selected a single school per student ignoring student mobility. We 
also evaluated an estimation of the random variance at the school level for each model in order 
to ensure that the model would converge and have sufficient sample at the school level. 
Table B1 


LA’s BEST: Logit Model Fit Assessment — Pooled Cohorts Graduation/Completion 


Random error variance (%) 


Grade band Model DIC Elementary Middle 
Elementary only First school (1) 11747 0.16 (5%) na 
MMMC (2) 11739 0.17 (5%) na 
Middle only First school (3) 11693 na 0.17 (5%) 
MMMC (4) 11664 na 0.15 (4%) 
Elementary and middle First school (5) 11677 0.05 (2%) 0.15 (4%) 
MMMC (6) 11652 0.05 (1%) 0.19 (5%) 


Note. Random error variance percentages are estimated using the equation: (ICC=var(uo;)/(var(Uoj))+(72 / 3)). 


The DIC takes into account the complexity of the model, and for both outcome 
measures it was determined that Model 6, which used the MMMC approach at both 
elementary and middle school levels produced a better fit. In addition, our evaluation of the 
random error variance at the school level, the ESS for the important independent variables, and 
the accuracy diagnostics suggested no reasons to reject this model. As a result, Model 6 was 
used to address the research questions. Table B1 displays the model assessment results for 


each of six models we considered for the analysis of graduation/completion. 


A Benjamini-Hochberg (1995) approach was also used to control for false discovery on 
all primary analyses, which examined persistence (dropout) and graduation/completion for the 


pooled samples. Table B2 presents the corrections for the eight primary analyses. 
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Table B2 


LA’s BEST: Benjamini-Hochberg Correction for the Primary Analyses of Pooled Outcomes 


Samples Clustering p-value New critical Finding Statistical 
corrected rank (x) p-value p-value < new significance 
p-value (p,) (p, = 0.05x/8) critical p-value? after BH 
(Dx < px) correction 
Dropout: High attendance group 0.003 1 0.006 Yes Yes 
Graduation/Completion: High attendance group 0.003 2 0.013 Yes Yes 
Dropout: Low attendance group 0.016 3 0.019 Yes Yes 
Graduation/Completion: LA’s BEST combined (any dosage) 0.071 4 0.025 No No 
Graduation/Completion: Moderate attendance group 0.180 5 0.031 No No 
Graduation/Completion: Low attendance group 0.291 6 0.038 No No 
Dropout: Moderate attendance group 0.654 7 0.044 No No 
Dropout: LA’s BEST combined (any dosage) 0.691 8 0.050 No No 
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Appendix C: Sample Demographics by Dosage Group 


Table C1 


Demographics of the Cohort 1 LA’s BEST and Control Group Students: Before and After Matching (Low 


Attendance) 


Before matching 


After matching 


LA’s BEST Control LA’s BEST Control 
Student characteristics (N = 487) (N = 4,478) (n = 410) (n = 410) 
Race/Ethnicity 
Hispanic (%) 83.2 86.6 84.4 87.8 
Black (%) 11.7 5.0 10.0 7.1 
Asian (%) 1.2 3.2 1.5 1.7 
White (%) 2.7 3.0 2.7 2.2 
Other (%) 1.2 2.2 1.4 1.2 
Special programs status 
Poverty (%) 95.3 94.8 95.1 95.1 
English language learner (%) 59.1 66.4 62.0 62.0 
Redesignated fluent English proficient (%) 0.6 0.6 0.5 0.5 
Special education (%) 6.6 5.9 5.4 5.4 
Gifted (%) 2.1 2.1 2.0 1.7 
Student achievement 
Mean second grade ELA score -0.199 -0.048 -0.185 -0.187 
Mean second grade mathematics score -0.160 0.014 -0.148 -0.136 
Other characteristics 
Female (%) 50.3 49.4 48.7 48.7 
Average school attendance 151.5 156.8 153.7 154.0 
Mean behavior rating (1-4) 2.9 3.1 3.0 3.0 
School: Poverty (%) 92.1 91.7 92.3 92.0 
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Table C2 


Demographics of the Cohort 1 LA’s BEST and Control Group Students: Before and After Matching 


(Moderate Attendance) 


Student characteristics 


Race/Ethnicity 
Hispanic (%) 
Black (%) 
Asian (%) 
White (%) 
Other (%) 
Special programs status 
Poverty (%) 
English language learner (%) 
Redesignated fluent English proficient (%) 
Special education (%) 
Gifted (%) 
Student achievement 
Mean second grade ELA score 
Mean second grade mathematics score 
Other characteristics 
Female (%) 
Average school attendance 
Mean behavior rating (1-4) 
School: Poverty (%) 


Before matching 


After matching 


LA’s BEST 
(N = 553) 


83.5 
9.0 
2.0 
2.7 
2.8 


92.2 
58.0 
0.5 
6.1 
1.8 


-0.095 
0.005 


51.2 
156.0 
3.0 
91.4 


Control 
(N = 4,478) 


86.6 
5.0 
3.2 
3.0 
2.2 


94.8 
66.4 
0.6 
5.9 
2.1 


-0.048 
0.014 


49.4 
156.8 
3.1 
91.7 


LA’s BEST 
(n = 462) 


84.6 
8.2 
1.7 
2.8 
2.7 


92.4 
60.0 
0.2 
4.8 
1.7 


-0.049 
0.045 


50.4 
157.5 
3.1 
91.8 


Control 
(n = 462) 


85.1 
5.6 
2.2 
4.5 
2.6 


94.8 
60.0 
0.2 
4.8 
1.7 


-0.056 
0.024 


50.4 
156.4 
3.1 
91.8 
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Table C3 


Demographics of the Cohort 1 LA’s BEST and Control Group Students: Before and After Matching (High 


Attendance) 


Student characteristics 


Race/Ethnicity 
Hispanic (%) 
Black (%) 
Asian (%) 
White (%) 
Other (%) 
Special programs status 
Poverty (%) 
English language learner (%) 
Redesignated fluent English proficient (%) 
Special education (%) 
Gifted (%) 
Student achievement 
Mean second grade ELA score 
Mean second grade mathematics score 
Other characteristics 
Female (%) 
Average school attendance 
Mean behavior rating (1-4) 
School: Poverty (%) 


Before matching 


After matching 


LA’s BEST 
(N = 580) 


85.0 
5.0 
5.5 
3.4 
1.1 


94.0 
64.1 
0.3 
5.3 
2.1 


-0.042 
0.062 


54.7 
159.2 
3.1 
91.2 


Control 
(N = 4,478) 


86.6 
5.0 
3.2 
3.0 
2.2 


94.8 
66.4 
0.6 
5.9 
2.1 


-0.048 
0.014 


49.4 
156.8 
3.1 
91.7 


LA’s BEST 
(n =512) 


85.2 
4.9 
5.9 
2.9 
1.1 


93.9 
65.8 
0.4 
4.1 
2.3 


-0.005 
0.085 


54.3 
160.5 
3.1 
91.2 


Control 
(n =512) 


87.3 
5.3 
3.9 
2.0 
1.5 


96.9 
65.8 
0.4 
4.1 
2.5 


-0.018 
0.071 


54.3 
159.5 
3.1 
91.6 
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Table C4 


Demographics of the Cohort 1 LA’s BEST and Control Group Students: Before and After Matching 
(Interrupted Treatment) 


Before matching After matching 
LA’s BEST Control LA’s BEST Control 
(N=1,440) (N=4,478) (n=1,218) (n= 
Student characteristics 1,218) 
Race/Ethnicity 
Hispanic (%) 86.8 86.6 87.9 88.1 
Black (%) 7.6 5.0 6.8 5.0 
Asian (%) 2.1 3.2 2.1 2.3 
White (%) 2.2 3.0 1.8 2.6 
Other (%) 1.3 2.2 1.4 2.0 
Special programs status 
Poverty (%) 95.6 94.8 96.0 94.8 
English language learner (%) 62.7 66.4 66.0 66.0 
Redesignated fluent English proficient (%) 0.6 0.6 0.3 0.3 
Special education (%) 4.9 5.9 3.0 3.0 
Gifted (%) 1.3 2.1 1.4 1.5 
Student achievement 
Mean second grade ELA score -0.197 -0.048 -0.167 -0.157 
Mean second grade mathematics score -0.135 0.014 -0.122 -0.105 
Other characteristics 
Female (%) 50.4 49.4 50.1 50.1 
Average school attendance 156.2 156.8 156.9 156.4 
Mean behavior rating (1-4) 3.0 3.1 3.0 3.1 
School: Poverty (%) 92.0 91.7 92.3 91.9 
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Table C5 


Demographics of the Cohort 2 LA’s BEST and Control Group Students: Before and After Matching (Low 


Attendance) 


Student characteristics 


Race/Ethnicity 
Hispanic (%) 
Black (%) 
Asian (%) 
White (%) 
Other (%) 
Special programs status 
Poverty (%) 
English language learner (%) 
Redesignated fluent English proficient (%) 
Special education (%) 
Gifted (%) 
Student achievement 
Mean second grade ELA score 
Mean second grade mathematics score 
Other characteristics 
Female (%) 
Average school attendance 
Mean behavior rating (1-4) 
School: Poverty (%) 


Before matching 


After matching 


LA’s BEST 
(N = 539) 


87.9 
7.4 
1.9 
0.6 
2.2 


89.4 
56.8 
5.6 
6.7 
2.2 


-0.172 
-0.084 


53.1 
157.6 
3.0 
86.8 


Control 
(N = 4,110) 


86.4 
5.0 
3.4 
3.2 
2.0 


90.7 
59.6 
6.4 
5.5 
2.5 


-0.038 
0.062 


48.4 
162.0 
3.1 
87.5 


LA’s BEST 
(n = 480) 


89.6 
6.5 
2.1 
0.4 
1.4 


89.4 
59.2 
4.6 
4.6 
2.5 


-0.145 
-0.038 


53.5 
159.2 
3.1 
86.7 


Control 
(n = 480) 


88.1 
4.4 
1.9 
3.8 
1.8 


92.3 
59.2 
4.6 
4.6 
2.5 


-0.140 
-0.028 


53.5 
160.1 
3.1 
87.4 
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Table C6 


Demographics of the Cohort 2 LA’s BEST and Control Group Students: Before and After Matching 
(Moderate Attendance) 


Before matching After matching 
LA’s BEST Control LA’s BEST Control 
Student characteristics (N = 580) (N = 4,110) (n =519) (n= 519) 

Race/Ethnicity 

Hispanic (%) 84.5 86.4 85.0 85.4 

Black (%) 8.1 5.0 7.3 6.9 

Asian (%) 2.9 3.4 3.1 1.7 

White (%) 3.1 3.2 3.3 4.2 

Other (%) 1.4 2.0 1.3 1.8 
Special programs status 

Poverty (%) 88.6 90.7 88.6 90.6 

English language learner (%) 50.0 59.6 51.6 51.6 

Redesignated fluent English proficient (%) 5.3 6.4 5.6 5.6 

Special education (%) 6.2 5.5 3.7 3.7 

Gifted (%) 1.7 2.5 1.9 3.1 
Student achievement 

Mean second grade ELA score -0.073 -0.038 -0.031 -0.013 

Mean second grade mathematics score -0.052 0.062 -0.012 -0.005 
Other characteristics 

Female (%) 53.6 48.4 52.8 52.8 

Average school attendance 160.7 162.0 162.3 161.7 

Mean behavior rating (1-4) 3.1 3.1 3.1 3.1 

School: Poverty (%) 85.9 87.5 86.0 87.1 
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Table C7 


Demographics of the Cohort 2 LA’s BEST and Control Group Students: Before and After Matching (High 
Attendance) 


Before matching After matching 
LA’s BEST Control LA’s BEST Control 
Student characteristics (N =486) (N=4,110) (n = 426) (n = 426) 
Race/Ethnicity 
Hispanic (%) 82.5 86.4 82.2 83.8 
Black (%) 7.8 5.0 7.7 6.3 
Asian (%) 4.9 3.4 5.4 4.7 
White (%) 2.1 3.2 1.6 2.8 
Other (%) 0.0 2.0 0.0 0.0 
Special programs status 
Poverty (%) 88.3 90.7 88.7 90.6 
English language learner (%) 48.6 59.6 52.1 52.1 
Redesignated fluent English proficient (%) 7.8 6.4 7.7 7.7 
Special education (%) 4.9 5.5 3.1 3.1 
Gifted (%) 3.1 2.5 3.1 3.8 
Student achievement 
Mean second grade ELA score 0.060 -0.038 0.090 0.099 
Mean second grade mathematics score 0.106 0.062 0.139 0.133 
Other characteristics 
Female (%) 55.8 48.4 54.9 54.9 
Average school attendance 167.2 162.0 168.7 166.3 
Mean behavior rating (1-4) 3.1 3.1 3.2 3.2 
School: Poverty (%) 85.8 87.5 86.0 87.5 
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Table C8 


Demographics of the Cohort 2 LA’s BEST and Control Group Students: Before and After Matching 


(Interrupted Treatment) 


Before matching 


After matching 


LA’s BEST Control LA’s BEST Control 
(N=1,313) (N=4,110) (n=1,160) (n= 
Student characteristics 1,160) 
Race/Ethnicity 
Hispanic (%) 85.6 86.4 86.6 86.8 
Black (%) 8.1 5.0 6.7 5.0 
Asian (%) 2.7 3.4 2.9 3.3 
White (%) 1.7 3.2 1.6 3.1 
Other (%) 1.9 2.0 2.2 1.8 
Special programs status 
Poverty (%) 91.6 90.7 91.6 91.0 
English language learner (%) 59.5 59.6 62.2 62.2 
Redesignated fluent English proficient (%) 4.9 6.4 4.3 4.3 
Special education (%) 6.3 5.5 4.7 4.7 
Gifted (%) 1.9 2.5 2.2 1.4 
Student achievement 
Mean second grade ELA score -0.211 -0.038 -0.201 -0.188 
Mean second grade mathematics score -0.066 0.062 -0.037 -0.051 
Other characteristics 
Female (%) 45.8 48.4 45.5 45.5 
Average school attendance 159.8 162.0 161.4 161.6 
Mean behavior rating (1-4) 2.9 3.1 3.0 3.0 
School: Poverty (%) 87.2 87.5 87.2 87.9 
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Appendix D: Model Coefficients 


Table D1 


Secondary School Dropout Model Coefficients 


Variables 


LA’s BEST & control group dummy codes 
Low attendance: Control 
Low attendance: LA’s BEST 
Moderate attendance: Control 
Moderate attendance: LA’s BEST 
High attendance: Control 
High attendance: LA’s BEST 
Stopped attending: LA’s BEST 
Race/Ethnicity 
Hispanic 
Black 
Special programs status 
English language learner 
Redesignated fluent English proficient 
Special education 
Gifted 
Student achievement 
Mean second grade ELA score 
Mean second grade mathematics score 
Other characteristics 
Female 
Second grade school attendance 
Poverty 
Mean behavior rating (1-4) 


School: Poverty 
*o s.05. 


Model coefficients (SD) 


Pooled 


-0.149 (0.106) 
0.150 (0.098) 
-0.044 (0.103) 
-0.099 (0.103) 
0.014 (0.105) 
-0.395 (0.118)* 
0.057 (0.075) 


0.356 (0.131)* 
0.790 (0.169)* 


0.003 (0.065) 
0.449 (0.223)* 
-0.281 (0.138)* 
-0.054 (0.239) 


-0.110 (0.050)* 
0.070 (0.048) 


-0.181 (0.057)* 
-0.002 (0.001)* 
0.159 (0.116) 
-0.309 (0.050)* 
-0.001 (0.003) 
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Cohort 1 


-0.161 (0.156) 
0.070 (0.149) 
-0.110 (0.150) 
-0.448 (0.167)* 
0.017 (0.142) 
-0.329 (0.155)* 
0.266 (0.103)* 


0.460 (0.194)* 
0.781 (0.242)* 


-0.042 (0.094) 
0.727 (0.600) 
-0.108 (0.190) 
-0.474 (0.428) 


-0.137 (0.073) 
-0.127 (0.069) 


-0.224 (0.078)* 
-0.002 (0.001) 
0.098 (0.184) 
-0.222 (0.070)* 
-0.007 (0.008) 


Cohort 2 


-0.176 (0.151) 
0.230 (0.141) 
0.005 (0.143) 
0.135 (0.142) 
0.004 (0.155) 

-0.513 (0.178)* 

-0.209 (0.113) 


0.382 (0.183)* 
1.058 (0.232)* 


0.059 (0.092) 
-0.584 (0.250) 
-0.476 (0.205) 

0.174 (0.306) 


-0.100 (0.070) 
0.006 (0.067) 


-0.135 (0.082) 
-0.002 (0.001) 
0.193 (0.149) 
-0.417 (0.072)* 
0.000 (0.004) 


Table D2 


High School Graduation/Completion Model Coefficients 


Variables 


Model coefficients (SD) 


Pooled 


Cohort 1 


Cohort 2 


LA’s BEST & control group dummy codes 
Low attendance: Control 
Low attendance: LA’s BEST 
Moderate attendance: Control 
Moderate attendance: LA’s BEST 
High attendance: Control 
High attendance: LA’s BEST 
Stopped attending: LA’s BEST 
Race/Ethnicity 
Hispanic 
Black 
Special programs status 
English language learner 
Redesignated fluent English proficient 
Special education 
Gifted 
Student achievement 
Mean second grade ELA score 
Mean second grade mathematics score 
Other characteristics 
Female 
Second grade school attendance 
Poverty 
Mean behavior rating (1-4) 


School: Poverty 
*o <.05. 


0.001 (0.090) 
-0.114 (0.090) 
0.055 (0.089) 
0.200 (0.091)* 
0.011 (0.091) 
0.346 (0.098)* 
-0.143 (0.067)* 


-0.414 (0.116)* 
-0.634 (0.155)* 


-0.043 (0.055) 
0.444 (0.193)* 
0.416 (0.115)* 

-0.212 (0.204) 


0.214 (0.044)* 
0.248 (0.041)* 


0.310 (0.048)* 
0.003 (0.001)* 
-0.248 (0.101)* 
0.560 (0.044)* 
0.001 (0.003) 
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-0.009 (0.131) 
-0.009 (0.130) 
0.226 (0.130) 
0.426 (0.136)* 
0.005 (0.125) 
0.384 (0.130)* 
-0.249 (0.090)* 


-0.447 (0.164)* 
-0.712 (0.213)* 


-0.016 (0.079) 
-0.449 (0.604) 
0.182 (0.158) 
0.037 (0.331) 


0.236 (0.063)* 
0.276 (0.060)* 


0.307 (0.069)* 
0.003 (0.001)* 
-0.217 (0.163) 
0.504 (0.061)* 
0.005 (0.007) 


0.051 (0.126) 
-0.203 (0.127) 
-0.090 (0.126) 

0.041 (0.127) 

0.029 (0.138) 
0.349 (0.147)* 

0.012 (0.098) 


-0.505 (0.158)* 
-0.745 (0.213)* 


-0.062 (0.077) 
0.532 (0.211)* 
0.650 (0.168)* 
-0.349 (0.267) 


0.221 (0.061)* 
0.191 (0.059)* 


0.314 (0.073)* 
0.003 (0.001)* 
-0.270 (0.129)* 
0.632 (0.064)* 
0.001 (0.003) 
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